Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms

The paper proposes a general framework which encompasses the training of neural networks and the adaptation of filters. We show that neural networks can be considered as general non-linear filters which can be trained adaptively, i. e. which can undergo continual training with a possibly infinite number of time-ordered examples. We introduce the canonical form of a neural network. This canonical form permits a unified presentation of network architectures and of gradient-based training algorithms for both feedforward networks (transversal filters) and feedback networks (recursive filters). We show that several algorithms used classically in linear adaptive filtering, and some algorithms suggested by other authors for training neural networks, are special cases in a general classification of training algorithms for feedback networks. INTRODUCTION The recent development of neural networks has made comparisons between "neural" approaches and classical ones an absolute necessity, in order to assess unambiguously the potential benefits of using neural nets to perform specific tasks. These comparisons can be performed either on the basis of simulations which are necessarily limited in scope to the systems which are simulated or on a conceptual basis endeavouring to put into perspective the methods and algorithms related to various approaches. The present paper belongs to the second category. It proposes a general framework which encompasses algorithms used for the training of neural networks and algorithms used for the estimation of the parameters of filters. Specifically, we show that neural networks can be used adaptively, i.e. can undergo continual training with a possibly infinite number of time-ordered examples in contradistinction to the traditional training of neural networks with a finite number of examples presented in an arbitrary order; therefore, neural networks can be regarded as a class of non-linear adaptive filters, either transversal or recursive, which are quite general because of the ability of feedforward nets to approximate non-linear functions. We further show that algorithms which can be used for the adaptive training of feedback neural networks fall into four broad classes; these classes include, as special instances, the methods which have been proposed in the recent past for training neural networks adaptively, as well as algorithms which have been in current use in linear adaptive filtering. Furthermore, this framework allows us to propose a number of new algorithms which may be used for non-linear adaptive filtering and for non-linear adaptive control. The first part of the paper is a short presentation of adaptive filters and neural networks. In the second part, we define the architectures of neural networks for non-linear filtering, either transversal or recursive; we introduce the concept of canonical form of a network. The third part is devoted to the adaptive training of neural networks; we first consider transversal filters, whose training is relatively straightforward; we subsequently consider the training of feedback networks for non-linear recursive adaptive filtering, which is a much richer problem; we introduce undirected, semi-directed, and directed algorithms, and put them into the perspective of standard approaches in adaptive filtering (output error and equation error approaches) and adaptive control (parallel and seriesparallel approaches), as well as of algorithms suggested earlier for the training of neural networks.